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Introduction 



This third report of the Second National Study of ESEA Title 
III has grown out of a need for more comprehensive planning of 
PACE. Ideally, the effort should have been made two years ago, but 
we did not know enough then. Hopefully, it will still be of some value. 

Responsibility for development of the model rests with the di- 
rector. Criticism or credit should be his, as the effort was not a 
group activity of the study team. I would like, however, to express my 
appreciation to those who did react to the document and thereby im- 
prove it. 

This study is sponsored by an ESEA Title III grant to the 
Center for Effecting Educational Change, Fairfax County, Virginia, 
which subcontracted the assignment to the University of Kentucky's 
Research Foundation. 



Richard I. Miller 
Director of Study 
November 10, 1968 
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A Comprehensive Model for 
Managing an ESEA Title III Project 
From Conception to Cnlmination 

The concept of total planning is just making its debut in educa- 
tion, in fact, the techniques for this procedure, growing out of mili- 
tary and industrial needs, are barely ten years old. The need for a 
more comprehensive approach to educatioixal planning is obvious in 
education as projects (a) become more complex, (b) become larger, 
(c) call for greater allocations of money, and (d) continue for a 
longer period of time. 

PACE has served to dramatize the glaring need that exists in 
United States education for better planning — a need that is not confined 
to this nation as indicated by a recent publication by the Organization 
for Economic Cooperation and Development, entitled, OECD and 
Educational Planning and Development . 

Those who have carefully studied ESEA Title III report some 
improvement in project quality as a result of the guideline require- 
ments for planning and as a result of several models for planning and 
evaluation. 

Most prominent among these models are the "CIPP" Model by 
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Daniel Stufflebeam,”^ a systems model by Donald Miller and Opera- 

V) / ^ / 

tion PEP,—' an evaluation model by Robert Hammond,— and a systems 

d / 

ctnalysis for self- ovciliicition. These efforts constitute recent and en- 
couraging efforts to bring more systematic planning and evaluation 
into education. 

One might ask: Why yet another model? Perhaps two answers 

are relevant: (1) We are just beginning to approach PACE in an 

orderly fashion and therefore all kinds of approaches are necessary 
in order that further sifting can take place; and (2) none of the pres- 
ent models is comprehensive. In other words, they do not apply to 
every conceivably important phase of the project — from its inception 
to termination. Such a. comprehensive approach could facilitate more 

effective operation by: 



—^Daniel L. Stufflebeam is director of The Evaluation Center at 
The Ohio State University, and he is special advisor for this national 
study. Description of CIPP may be obtained by writing him at the 
Evaluation Center. 

—^Donald R. Miller is director of Operation PEP, Preparation 
for Educational Planners, Burlingame, California, and he is a mem- 
ber of the project directors' advisory group for this study. Address: 
1870 El Camino Real, Burlingame, California 94010. 

—^Robert L. Hammond was director of Project EPIC, Evalua- 
tive Programs for Innovative Curriculums, Tucson, Arizona, and he 
is a member of the project directors' advisory group for this study. 
Address: 1034 East Adams, Tucson, Arizona 85719. Dr. Hammond 

is now with the Evaluation Center at The Ohio State University. 

— ^Edward J. Ambry, director. The New Jer sey State Council 
for Environment Education, Board of Education, Newark, New Jersey. 
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1. Assisting all parties concerned with a PACE proposal 

in tying their efforts together to the total purposes 
and thrusts of the particular project. These parties 
are: proposal writers, project operators, project 

evaluators (inside and outside), and state and federal 
officials . 

2. Assisting those involved most directly in each of the 
subsequently mentioned segments in systematically 
considering their tasks. 

3. Providing a procedure for evaluating projects that 
will be more likely to result in self adjustment and 
improvement by providing useful feedback data. 

4. Assisting educators to think more systematically 
about their concerns. 

This model should be considered suggestive rather than pre- 
scriptive. The user is asked to consider each variable listed but he 
should not be expected to find all of them applicable. The variables, 
however, should serve as reminders of points that may be relevant. 
One should not view this model as a limitation upon freedom of choice 
rather, as ar. expansion of it. Points raised in this model might not 
have been considered otherwise and therefore the project director’s 
freedom is increased by having a wider range of alternatives. 

The model is composed of five sections, each of which has 
numerous subsections: 

1. 0 PROPOSAL DEVELOPMENT (yellow section) 

2. 0 FIRST year APPRAISAL (blue section) 

3. 0 SECOND YEAR APPRAISAL (red section) 

4. 0 THIRD YEAR APPRAISAL (green section) 

5. 0 FINAL APPRAISAL (white section) 



o 
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U ses of the Model 

This model is designed to assist all phases of a PACE pro- 
ject. As such, it could be useful for (a) those who write projects, 

(b) those who operate projects, and (c) those who evaluate projects,, 
both in terms of general improvements and in terms of continuation. 

Officials who approve and evaluate ongoing projects need a 
standard against which to make their suggestions and judgments. The 
first phase — proposal development — should provide this yardstick. 
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Characteristics of Project 



Project Characteristics 

1. Grant number 

2. Period of project (first, second, third year) /_circle one/ 

3. Total amount of grant 

4. ^Non-Federal support 

5. Federal support under Title III 

6. , Federal support other than Title III 

7. Major description of project /Fheck one onl^ 

a. Innovative 

^b. Exemplary 

c. Adaptive 

8. Types of activity /_check one or more? 

a. Planning of program 

^b. Planning of construction 

c. Conducting pilot activities 

^d. Operation of program 

e. Constructing 

f. Remodeling 

9. Project title [5 words of less? 



10, State of applicant 

11. Scope of project /pheck one onl‘£7 
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r 



a. 

b. 



c. 

d. 



e. 

f. 

.g- 

h. 



National 

Regional (multi- state) 

One state 
One district 

Multi-district; meaning two to six districts 

Multi-district (or county); meaning seven 
or more districts within one state 

A few schools within one district 

One school 



12. Area served by project /jDest fit on^ 

a. Urban 



(1) central city 

(2) urban fringe 

b. Other urban (suburban) 

(1) cities of 10, 000 or more 

(2) cities of 2, 500 to 10, 000 

c. Rural 

(1) places of 1, 000 to 2, 500 

(2) other rural 




i 

if 




1. PROPOSAL DEVELOPMENT 



ESEA Title III projects begin basically in one of two ways: 

Either the nature of the idea or program is clear and compelling and 
a systematic needs assessment is not necessary (usually in the case 
of a single idea or program), or an extensive needs assessment is 
undertaken to establish priorities and program (usually in the case of 
a comprehensive or supplementary center — type of project). 

Application of the model proposal outlined in this report will be 
somewhat different in each case. If focus is on a single idea or pro- 
gram, those developing the proposal may want to "plug in" at the . . 

1.4. 2. 2 (objectives section) skipping over the material on needs assess- 
ment. 

The proposal development phase (1.) suggests these four major 
sections and subsections, and in this order: 1. 1 task forcej 1. 2 needs 

assessment; 1.3 priorities; and 1.4 proposal development. 

1 . 1 Task F or ce 

The first step in a PACE proposal should be formulation of a 
task force that will serve to assess needs and write the subsequent 
proposal. This body should be an ad hoc group that may or may not go 
out of existence when the proposal is either approved or rejected. 
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depending primarily upon the local situation and the committee's ef- 
fectiveness. An ad hoc status provides the authorities with some 
flexibility in this matter. 

The following considerations may be relevant to the formation 



of an effective task force. 



1 . 1 . 1 



What should be the membership? 

1. 1. 1. 1 Administrators 

1 . 1 . 1 . 2 Teachers 

1. 1. 1. 3 Academic scholars 

1. 1. 1.4 Professors of education 

1. 1. 1.5 Members of the community, including (a) 
persons broadly representative of the cul- 
tural and educational resources of the areas 
to be served, and (b) persons representa- 
tive of the interests of potential beneficiaries 



1 . 1 . 1.6 Students 

1. 1. 1. 7 Other 



1.1.2 What tasks should ad hoc group undertake? 

1.1.3 Are human and material resources compatible with 
the chosen parameters? 

1. 1.4 Is the time allottment realistic in terms of chosen 
parameters? 

1.1.5 What is the anticipated cost? 

1. 1. 5. 1 Is financial support adequate for the chosen 
tasks? 

1. 1. 5. 2 Is adequate money set aside for evaluation? 



1 . 2 Needs Assessment 

Many needs assessment are an endless accumulation of sta- 
tistics with little delineation of the area's basic nature and needs. A 
successful needs assessment usually is a time consuming and com- 
plex task but well worth the effort in terms of providing a basis for 



o 
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action and for meaningful community participation and support. A 



ne 



eds assessment study might consider this pattern. 



1.2.1 Obiectives 

1.2. 1. 1 What are the objectives of the assessment? 

To determine socio-economic 
area needs, as viewed by 
selected audiences. 

To define greater education 
needs of the area, as viewed 
by selected audiences. 

To determine where future 
educational thrusts should be 
made . 

To determine priorities. 
Other 

Are needs assessment objectives phrased 
in measurable terms? 



1 . 2 . 1 . 1 . 1 



1 . 2 . 1 . 1.2 



1.2. 1. 1.3 



1 

1 



2 

2 



1. 1.4 
1.1.5 



1 . 2 . 1 . 2 



1.2.2 Who will determine needs ? 

1. 2. 2. 1 Members of the present staff (who) 

1. 2. 2. 2 New personnel (where available) 

1.2. 2. 3 Special study 

1.2. 2. 4 "Outside" consultants 

1. 2. 2. 5 Members of the community 

1 . 2. 2. 6 Students 

1.2. 2. 7 Other 



1.23 What ar e the demographic chara cteristics? 

1.2.3. 1 Population 

1.2.3. 1. 1 Past 

1.2.3. 1.2 Present 

1.2. 3. 1.3 Future projections 

1.2. 3. 2 Race distribution 

1.2. 3. 3. Age distribution 

1.2. 3. 4 Sex distribution 

1. 2. 3. 5 Employment breakdown 

1.2. 3. 5.1 Professional 



1. 2. 3. 5. 2 

1. 2. 3. 5. 3 

1. 2. 3. 5. 4 

1.2. 3. 5. 5 

1. 2. 3. 5. 6 

1. 2. 3. 5. 7 



Managerial 

Clerical 

Sales 

Craftsmen 

Operatives 

Service 
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1. 2. 3. 5. 8 Farmers 

1.2. 3. 5. 9 Laborers, except farm, but 
including private household 
help 



1.2.4 What resources in the area may be relevant ? 



1.2. 4. 1 


Human 






1.2.4. 1. 1 


Specialists 




1.2. 4. 1.2 


Consultants 




1. 2.4. 1. 3 


Other 


1.2. 4. 2 


Material 






1. 2. 4. 2. 1 


Natural 




1. 2.4. 2. 2 


Building s 




1. 2. 4. 2. 3 


Equipment 




1. 2. 4. 2. 4 


Other 


1.2. 4. 3 


Financial 






1. 2.4. 3. 1 


Business and industry 




1. 2.4. 3. 2 


Labor 




1. 2.4. 3. 3 


Private philanthropy 




1.2. 4. 3. 4 


Other 



1.2. 5 



What are 
1.2. 5. 1 

1.2. 5. 2 

1.2. 5. 3 
1.2. 5.4 
1. 2. 5, 5 
1.2. 5. 6 

1. 2. 5. 7 



1.2. 5. 8 

1. 2. 5. 9 
1. 2. 5. 10 



the educational patterns ? 

What student achievement measures, such 
as I.Q. and other measurements, are used? 
What is the socio-economic class structure 
of school system? 

What is the composition of student body? 

What are the teacher salaries? 

What is the teacher turnover? 

Are the high school courses related to the 
occupation makeup of the area? 

Are the high school courses related realis- 
tically to projected occupational and growth 
patterns for the area? 

How do the schools "rate" with others in the 
state? Do you have criteria for making such 
judgment? 

What general evidences of progressiveness 
are evident in the school system? 

Have innovations (new programs) been intro- 
duced recently in the regular subjects, in 
individualization, in instruction, and/or in 
organization (e. g. , non-gradedness, team 
teaching)? 
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What evaluation is pla: 


nned for needs assessment? 


1. 2. 6. 1 


Who is to evaluate? 


1. 2. 6. 2 


By using: 






1. 2. 6. 2. 1 


Anecdotal report 




1. 2. 6. 2. 2 


Checklist 




1. 2. 6. 2. 3 


Outside experts 




1. 2. 6. 2. 4 


Locally made tests 




1.2.6 2.5 


Standardized tests 




1. 2. 6. 2. 6 


Rating scales 




1. 2. 6. 2. 7 


Questionnaire 




1. 2. 6. 2. 8 


Research design (such as using 
pre-test and post-test) 




1. 2. 6. 2. 9 


Other evidences 



1. 3 Priorities 

The needs assessment study should produce a relatively small 
number of activities that have been found to be most important for 
schools in the region. At this point formal action should be taken on 
the priorities by the board of education or some other legally desig- 
nated body. The staff will need to have data available on a number of 
items. These questions might assist in the selection process. 

1.3.1 Of the identified needs, which ones have the great- 
est potential for your situation? 

1.3.2 What priorities will strike most diiectly at great- 
est needs? 

1. 3. 3 Are the priorities realistic in terms of state and 
national priorities? 

1.3.4 Are the priorities manageable in terms of time, 

resources, personnel that likely will be available? 

1 . 4 Proposal Development 

A plan for developing and writing the proposal should be charted 
at this time. Individuals on the needs assessment task force might be 
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used in the next steps — or perhaps different skills are needed. Time, 
personnel, and resources should be considered in developing the 
strategy for proposal development. 

1.4.1 A Plan for Proceeding 

At this point a general overview of the task of proposal 
development should be undertaken. Those who are "old hands" at pro- 
posal development realize the importance of careful consideration of a 
number of points at this time. 

1.4. 1. 1 How much time — working time — is avail- 
able for the proposal development? 

1.4. 1.2 What tasks need to be done? 

1.4. 1.3 Who should do them? 

1.4. 1.4 How much money will be needed? 

1.4.2 Contents of Proposal 

Proposal writers are caught in a perennial dilemma: 
How can the proposal have sufficient detail and organization yet not 
bind future operations to prenatal planning? The dilemma, , however, 
can be solved. The proposal should reflect a systematic approach, but 
all parties concerned should recognize the great probability that modi- 
fications will be needed as the proposal moves from words to action. 
Indeed, it is desirable that evaluative procedures take cognizance of 
the high probability of modifications. 

It seems reasonable to expect proposals to follow a 
systematic approach in describing what is being planned thereby pro- 
viding a yardstick against which evaluations can be made. The 
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following components are suggested as relevant to such a systematic 
approach: 

1.4.2. 1 Background 

This section relies primarily upon data 
from the needs assessment study as well as other data about the com- 
munity. If a needs assessment has not been made, then several points 
raised in that section (1.2) would be relevant here. 

1. 4. 2. 1. 1 Is background data relevant to 
the proposal? 

1. 4. 2. 1.2 Is data recent and accurate? 

1 . 4. 2. 1 . 3 If "needs assessment" data is 
used, are priorities spelled 
out with sufficient specificity 
for program development? 

1.4. 2. 2 Obi ectives 

Objectives are destinations — the targets — 
and they need to be determined at both general and specific levels. 
General objectives are about as far as most proposals go, but this is 
only one stage. The next stage — specific objectives — forces the appli- 
cant to say what the gains are sought in terms that can be measured or 
assessed . The objectives, then, become the measures against which 
program progress is appraised. Objectives - program - evaluation 
(basic components of a system analysis approach) become the basic 
components of the proposal and of all subsequent development. For 
this reason the careful formulation of objectives is essential. 
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1. 4. 2. 2. 1 What are the general objec- 
tives? 

1.4. 2. 2. 2 What are the specific objec- 
tives? 

1.4. 2. 2. 2.1 Deprograms 

grow out of the 
objectives? 

1.4. 2. 2. 2. 2 Are objectives 

stated in terms 
that will allow 
measurement of 
progress toward 
them? 

1. 4. 2. 2. 3 A final objective of most PACE 



projects is improving education for individual children and youth, 
which moans bringing about some positive change in knowledge, skills, 
and/or values of children and youth. Such projects are remiss in their 
evaluation if some systematic procedure — simple or complex — is not 
developed to measure impact upon children and youth in the target 
area. To this end, some specific student objectives, and accompanying 
measurement devices, may be suggested by these questions. 

. . 1.4. 2. 2. 3.1 Are positive 

changes in 
knowledge, 
skills and/ or 
youth important 
objectives? 

(Are these ob- 
jectives stated 
in terms that 
can be mea- 
sured? ) 
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1.4. 2. 2. 3. 2 What measure- 
ment devices 
are proposed 
for judging each 
student- pointed 
objective? 



1.4. 2. 3 Programs 

This section spells out what program or 
programs tlie project will seek to accomplish, and it is usually the 

most extensive part of the proposal. 

1. 4. 2. 3. 1 What programs are planned? 

1.4. 2. 3. 1.1 Are they stated 

in clear and con- 
cise language, 
so that the "un- 
initiated" can 
understand 
them? 

1. 4. 2. 3. 1. 2 Does the pro- 
posal attempt to 
do too much, as- 
suming that the 
designers want 
to do only as 
much as they 
can do well? 

1.4. 2. 3. 2 Were existing school curricula 
and programs reviewed in de- 
veloping the project's program? 

1.4. 2. 3. 3 Are the programs mxanageable 



in terms of; 
1.4. 2. 3. 3. 1 


Time allocations 


1.4. 2. 3. 3. 2 


for each seg- 
ment? 

Personnel need- 


1. 4. 2. 3. 3. 3 


ed? 

Resources need- 




ed to do the job 
properly? 



o 
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1. 4. 2. 3. 4 Are programs specifically re- 

lated to objectives? 

1. 4. 2. 3. 5 What procedures and techni- 

ques of communication and 
dissemination have been con- 
sidered, not in terms of ex- 
pected results of project but 
in terms of mustering support 
for the program? (See sub- 
section on dissemination in the 
section on "project operation.") 

1.4. 2. 4 Demonstration 



Is project or program really worth demon- 
strating? Is it the kind of activity that lends itself to demonstration? 
Sometimes too little thought goes into what should be demonstrated — 
and how. Also, sometimes little attention is given to what effects the 
demonstration will have upon children in the demonstration unit. But 
demonstration, properly conducted, remains a dynamic factor in the 
change process. 



1. 4. 2. 4. 1 
1.4. 2. 4. 2 

1. 4. 2. 4. 3 

1.4. 2.4.4 
1.4. 2.4. 5 



What will be demonstrated? 

1.4. 2.4. 1.1 By whom? 

Is demonstration the best way 
of dissemination in your in- 
stance? 

How will the demonstration be 
conducted? 

What will be the cost? 

How will the effects be 
evaluated? 



1.4. 2. 5 Dissemination 



Dissemination, in the simplest terms, is 



the spreading of an idea. Unfortunately, too little serious attention has 
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beeii given to this important link in the change process. Many PACE 
projects might well reconsider their approach to dissemination, and 
at three levels: (1) in terms of initial communication about the pro- 

ject — a general informing; (2) some occasional or regular medium 
for specific in'^orming; and (3) in terms of disseminating results of 
completed projects in a manner that will facilitate their implication in 
other schools. The first and second types of dissemination occur very 
shortly after the project becomes operational, and the third becomes 
relevant when something is achieved. 

All dissemination should follow some 
orderly approach, but the proposal at this developmental stage should 
not be expected to go into much detail. This will come later when the 
project becomes operational . 

In the initial proposal, one can expect some 
evidences of thinking about priorities-objectives- strategies-evaluation 
of dissemination. 



1.4. 2. 5. 1 



1.4. 2. 5. 2 
1.4. 2. 5. 3 



What kinds of dissemination 
should be employed for your 
project, and for what audi- 
ences? The following grid 
might be helpful in pinpointing 
some of these concerns. 

What information should be 
sent r 

What are the intended results 
from sending this information? 
1. 4. 2. 5. 3. 1 In terms of your 

program? 

In terms of tar- 
get audience? 



1.4. 2. 5. 3. 2 



} 
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Audiences 



Board of Education 
Colleges and Universities 
Community 

Local Administrative Staff 
PACE Controlling Board 
Project Staff 
Specializing Groups 
State Department of 
Education 
Students 
Teachers 



Mediums 























































































































































































Informal Meet- 
ings 


In-service Pro- 
grams 


Newspaper 


Radio 


Regular Meet- 
ings 


Regular News- 
letter 


Special Bulletins 
(Memos) 


Special Stories 


Television 



1.4. 2. 5. 4 

1.4. 2. 5. 5 

1.4. 2. 5. 6 
1.4. 2. 5. 7 



How much time will be requir- 
ed? 

How much money will be requir- 
ed? 

What personnel will be required? 
Evaluation? One of several 
procedures should be employed 
to determine whether the dis- 
semination has achieved stated 
objective£ for target audiences. 
1.4. 2. 5. 7.1 Survey (question- 
naire) 

1.4. 2. 5. 7. 2 Interview 
1.4.2. 5. 7. 3 Unsolicited feed- 
back (e. g. , let- 
ters, telephone 
calls, mass 
media news, in- 
formal com- 

m f»nt 



O 
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1.4. 2. 5. 8 



1. 4. 2. 5. 9 



1. 4. 2. 5. 7. 4 Research design 
1 . 4. 2 . 5. 7. 5 Other 
Is a system proposed for cata- 
loging, interpreting and distri- 
buting information to be de- 
veloped in the project? 

In judging content, the follow- 
ing criteria might be useful: 

1. 4. 2. 5. 9. 1 Clarity 

V alidity 
Pervasivenes s 
Impact 
Timeliness 
Practicality 



1. 4. 2. 5. 9. 2 
1. 4. 2. 5. 9. 3 
1. 4. 2. 5. 9. 4 
1.4. 2. 5. 9. 5 
1. 4. 2. 5. 9. 6 



1.4. 2. 6 Implementation 

Implementation is moving an idea into ac- 
tion through the use of planned strategy. It is, in essence, the "pay- 
off, " and it does not just happen — at least not ordinarily so. 

Educators tend to have three myths about 
implementation: (1) A good product will market itself, which is none- 

sense. Any industrial firm knows that a good project is a starting 
point, but a strong marketing function is essential also; (2) Everyone 
should be involved or at least give their blessing. This condition is 
idealistic and usually unobtainable. Implementation should be attempt- 
ed first with the converted. This group usually is challenge enough. 
(With some innovations, such as nongradedness, it may be necessary 
to have involved all teachers in a particular unit;) (3) Implementation 
of an idea or program requires little follow-up. We have a myth of 
perfectability, believing that introduction of the innovation is the thing. 



- 21 - 



Quite to the contrary, a "service department function" is vital to the 
continued growth and success of most innovations. 

Many strategies for implementing ideas are 
a/ 

available.— The purpose in this section is not to outline or to favor 
any one strategy*, rather, to help PACE projects think through some 
aspects that are common to most strategies for successful implemen- 
tation. Again, as with dissemination, the proposal should not be ex- 
pected to offer a detailed model for implementation, rather, one should 
expect evidences that implementation will be given adequate attention 
as the project develops. The following questions may help in more 
careful consideration of implementation: 

1 . 4. 2. 6. 1 Is the idea or program (inno- 
vation) exportable as it is, or 
must special adaptations be 
made? 

1. 4. 2. 6. 2 Is a timetable (plan) for ex- 
portation (implementation) of 
the idea developed? 

1. 4. 2. 6. 2. 1 Will a "service" 

function be pro- 
vided if expert 
advice may be 
needed? (e. g. , 
in terms of pro- 
grammed in- 
struction, non- 
gradednes s) 



a/ 

— ^Several strategies are listed in A Multidisciplinary F ocus 
on Educational Change . (Richard I. Miller, editor). Lexington, 
Kentucky: Bureau of School Services, University of Kentucky, 1964, 

p. 84. 
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1. 4. 2. 6. 3 Are recipient audiences care- 
fully selected? 

1. 4. 2. 6. 4 Who will serve as "change 

agents" — those most directly 
involved in bridging the gap 
between theory and action? 

1.4. . Terminal considerations 

After three years of Federal funding, it ap- 
pears at this writing that PACE projects will fold their tents and silently 
steal away. But let us hope that this is not the case, unless of course 
the project is not worth continuing— and there should be a few of these. 

Many projects, as they look now after two 
years of operation, are worthy of continuation. Since securing funds 
for continuation will not be an easy task in most cases, it is essential 
that planning for other funding should be initiated not later than the 
second year . 

Possibly one of the best ways of insuring 
PACE program continuity beyond Title III fuxxd availability is to sys- 
tematically structure the program activity into the regular district 
activity so that the program's clientele over the three years become 
so dependent upon the PACE thrust that prospects of its fund cutoff will 
help marshall other resources for its continuation. Some techniques 
for this systematic structuring are discussed in this paper, such as 
relationships, implementation, demonstration, and dissemination. 

Proposals should not be expected to give 
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evidence of more than passing interest in terminal considerations, but 
the first year appraisal should demonstrate consideration of the matter. 

1.4. 2. 8 Management 

The proposal should contain fairly detailed 

information about what kinds of personnel will be required, their job 
descriptions, and the overall chain of authority for the project— the 
line and staff structure. 



1. 4. 2. 8. 1 What special skills \vill be 
needed for the project? 

1.4. 2. 8. 1.1 Are fairly speci- 
fic job descrip- 
tions available 
for these posi- 
tions? (The key 
here is "fairly 
specific . " J ob 
descriptions 
should not be- 
come ends in 
themselves but 
means to de- 
velop a balanced 
staff. And some- 
times an outstand- 
ing person does 
not fit any job 
description. ) 

1. 4. 2. 8. 2 Do characteristics such as 
creative, innovative, open- 
minded, optimistic, intelligent, 
persevering, and humanly sen- 
sitive stand out in your search 
for personnel? 

1. 4. 2. 8. 3 Are provisions made for staff 
growth through in-service 
training and other means? 

1. 4. 2. 8. 3. 1 Is budget allocat- 
ed for this? 
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1.4. 2. 8. 4 
1.4. 2. 8. 5 



1.4. 2.8. 6 
1.4. 2. 8. 7 



1 . 4 . 2 . 8 . 8 



1 . 4 . 2 . 8 . 9 



What instrument or procedures 
are planned for personnel 
evaluation? 

What will be the organizational 
chart for the line and staff re- 
lationship betwe^' n these posi- 
tions? 

What procedures are provided 
for intrastaff communications? 
What technique or procedure is 
employed to plan for the most 
effective use of time, resources, 
and personnel over a period of 
time? (It may be a simple pro- 
cedure, such as a series of 
checkpoints at which thorough 
review will be niade of pro- 
gress, problems, and priori- 
ties; or it may be a more com- 
plicated technique such as 
PERT or the Ciitical Path.) 

Is adequate fiscal assistance 
for management functions pro- 
vided in the project? (Many 
projects have underestimated 
the amount of time required to 
keep abreast with regulations, 
red tape, and requisitions.) 

Is an organizational chart de- 
veloped? 



1.4. 2. 9 R elationships 

Every proposal should contain fairly specific 
information about both internal (teachers and administrators) and ex- 
ternal (community) relationships. Probably more PACE projects fa-j-l 
because of inadequate or poor relationships than from any other caus_e. 
This situation arises from a general lack of experience with proposal 
development. The tendency is to involve too many people in a 
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superficial way, laying the ground work for adverse reactions later. 



A number of questions should be raised: 



1. 4. 2. 9. 1 
1.4. 2. 9. 2 
1.4. 2. 9. 3 



1.4. 2. 9.4 

1.4.2. 9. 5 
1.4. 2. 9. 6 

1.4. 2. 9. 7 



Who must be involved for the 
sake of support? 

What is the purpose of any par- 
ticular involvement? 

How should the individual, 
group, or organization be in- 
volved? 

1. 4. 2. 9. 3. 1 Is the involve- 
ment busy work, 
or does it have 
potential rele- 
vance? 

When should the involvement 
take place? initially or through- 
out, and on a sporadic or regu- 
lar basis? 

Why is a particular involvement 
necessary? 

What evaluation procedures are 
planned to determine whether 
the various involvements are 
accomplishing their objectives? 
Is some sort of advisory com- 
mittee operative? (Periodic 
reporting and interacting with 
a carefully selected advisory 
committee can be most helpful 
to the project. Such a group 
can help prevent drifts toward 
failure as well as serve as a 
sounding board. The legally 
designated controlling board 
normally will not be as con- 
cerned with program develop- 
ment as the advisory group. ) 



1.4.2.10 Budget 



Budgets traditionally are placed at the end 
of proposals — although in practice the budget might be the first 
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consideration. And certainly budget considerations do permeate all 
phases of proposal development. 

The effectiveness of fiscal policies needs to 



be evaluated regularly, with subsequent adjustments of policy to meet 
the fiscal needs of the program as they are influenced by change in the 
financial structure and by changes in source and availability of funds. 

A number of strategies are developed with 
respect to "padding" budgets, and one can assume that some individuals 
are more clever at it than others, but generally speaking the budget 
should reflect what is essential for the program, with perhaps five per- 
cent leeway for contingencies. 



1 . 4 . 2 . 10 . 1 
1 . 4 . 2 . 10. 2 
1.4.2. 10. 3 

1.4.2. 10.4 

1. 4. 2, 10. 5 



Are standard bookkeeping pro- 
cedures to be used? 

Is the budget adequate to carry 
through the program? 

Are local and state fiscal pro- 
cedures thoroughly understood? 
Will funds be available for 
auditing financial records? 

Have the possibilities of secur- 
ing monies other than ESEA 
Title III been considered? 

1. 4. 2. 10. 5. 1 ESEA Title I 
1.4.2. 10. 5. 2 ESEA Title II 

1.4.2.10.5.3 ESEA Title IV 

1.4.2.10.5.4 Foundations 

1. 4. 2. 10. 5. 5 Private donors 
1. 4. 2. 10. 5. 6 Other 



1.4.2. 11 F acilities , equipment , and materials 

Careful consideration needs to be given to 
facilities, equipment, and materials — not only in terms of cost, but 



o 
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in terms of style, convenience, and appropriateness. 



1 . 4 . 2 . 11 . 1 
1.4. 2.11.2 
1 . 4 . 2 . 11. 3 
1.4.2. 11.4 

1.4. 2. 11. 5 



1 . 4 . 2 . 11. 6 
1.4.2. 11. 7 



What are basic facility needs 
for the project? 

Are these available when need- 
ed? 

If not, can rennovations be 
made? 

Have costs been checked out 
carefully, including costs of 
alternative plans? 

Does the proposal request 
adequate materials and sup- 
plies? 

1.4.2.11.5.1 Are sufficient 

funds available 
for these? 

1.4.2. 11. 5.2 What procedures 

will be used to 
select materials 
and supplies? 

1.4.2. 11.5.3 Will maintenance 

funds be avail- 
able? 

Will adequate lead time be 
given in placing the order? 

Does the project proposal give 
reasonable attention to an ef- 
ficient method for scheduling 
project personnel and students 
for use of available facilities 
and equipment? 



1.4.2.12 Evaluation 



Evaluation procedures should be related 
specifically to each program and to the objectives. Many educators 
still are mentally encased in traditional concepts of evaluation, such 
as pre- and post-tests, which do remain legitimate evaluative proce- 
dures for certain kinds of programs. Interviews, case studies, and 
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surveys are other examples of equally legitimate kinds of evaluative 
procedures. In other words, the methods of evaluation should suit the 



program and not vice-versa. 



1 . 4 . 2 . 12 . 1 



1 . 4 . 2 . 12. 2 

1.4. 2. 12. 3 



1.4. 2. 12.4 



Does the amount budgeted for 
evaluation account for a mini- 
mum of five percent of the over- 
all budget? (Some individuals 
frown upon a specific figure, 
and with good reason; however, 
the value of a specific figure 
lies in providing a minimal 
yardstick. Considering the 
overall picture, most benefits 
stand to be gained than lost 
from a specific figure. ) 

Is evaluation planned in an 
imaginative way? 

Which evaluative procedures 
are most appropriate? 
1.4.2.12.3.1 Anecdotal report 

Checklist 
Outside experts 
Standardized 
tests 

Rating scales 
Questionnaire 
Research design 
(such as using 
pre-test and 
post- test) 

Opinion 

Other evidences 
(e.g., willing- 
ness of schools 
to participate, 
to take over 
funding) 

What procedures or methods 
can be established for informal 
feedback on a more or less con- 
tinuous basis? (This proce- 
dure might involve a few key 



1 

1 

1 

1 

1 

1 



4 . 2 . 12 . 3. 2 
4 . 2 . 12 . 3. 3 
4. 2. 12. 3.4 

4.2.12.3.5 
4. 2. 12. 3. 6 
4 . 2 . 12 . 3. 7 



4 . 2 . 12 . 3. 8 
4 . 2 . 12 . 3. 9 
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individuals who serve as "eyes" 
and "ears;" they know the 
comnaunity very well and can 
spot trouble at the onset. The 
advisory committee might be 
useful here. ) 

1. 4. 2. 12. 5 Is evaluation in terms of how 

well the program is meeting its 
objectives? 

Irrespective of which evaluative procedure is used, the data 

should have those features commonly associated with good informa- 
a/ 

tion. — Four features suggested here: 

1. Internal validity . There should be a good, if not a one- 
for-one relationship, between the information and the phenomena it 
represents . 

2. External validity . Do the data apply only to the subjects 
from whom they were gathered, or may they be taken as representing 
what one might find if the evaluation were repeated elsewhere? In 
other words, under what circumstances are the evaluation data 
gener alizable? Considerations of generalizability have to do not only 
with whether or not the sample from which they were gathered was 
representative of some population, but they are also related to questions 
of physical arrangements, kinds of teachers, conditions under which 
data are collected, and so forth. 

3. Reliability . This criterion refers to the internal 



a/ 

— This section is based upon work by Egon Cuba. 
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consistency of the information. If it were gathered again, would the 
same findings be found? 

4. Objectivity . Is the evaluation based upon private evidence, 
such as the opinions of teachers, which, while perhaps replicable and 
therefore reliable, might not be in agreement with what would be found 
by an independent observer? Are the instruments used ca,pable of simi- 
lar interpretation by different but equally competent to judge agree on 
the meaning of the data? Objectivity is relating the answers to such 
questions. 

1.4.2. 13 Assurances 

The assurances constitute the various legal 
authorizations that provide the proposal with an official status. 



2. FIRST YEAR APPRAISAL 



With proposal approval the project becomes operational* and 
the initial proposal should serve as a roadmap for the journey toward 
intended accomplishments. 

At this point the staff basically may have, at the extreme, one 
of two experiences: Either the roadmap — the initial proposal—is 

reliable and useful, or it is the voice of Christmas past. The latter 
alternative may be the result of (1) developing a proposal with funding 
as the primary objective; (2) developing a poor proposal; (3) having 
weak project personnel; or (4) finding that new circumstances and/or 
factors have come upon the scene that were not anticipated when the 
proposal was developed. 

Six - Month' s R eport 

To deal with the possibility of some, or considerable, change in 
the project’s nature and direction, a mid-year checkpoint — a self-as- 
sessment — is suggested for every project. (See page 32.) The follow- 
ing procedure is suggested as a method of bringing project development 
into manageable perspective at the mid-year checkpoint. 

On one large sheet, a "gestalt" of the entire program should be 
developed. This overview can be quite helpful in providing perspective 
for the staff as well as others. 
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FIRST YEAR: SIX -MONTH REPORT 
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First Year Appraisal 

A careful appraisal should be made at the end of the first 
year, from two perspectives: (1) a self- appraisal by the project's 

staff, and (2) an appraisal by an outside group. The results of these 
two appraisals then should be considered by the state ESEA Title III 
coordinator in developing his recommendations for the state advisory 
council. 

The same instrument should be used for both internal and ex- 
ternal appraisals. The external appraisal group should make its 
evaluations independent of what the staff has done. 

The following two procedures are suggested for bringing the 
project into manageable perspective at the end of the first year. It is 
similar to the procedure recommended for the mid-year evaluation. 

The same schematic "gestalt" one-page picture suggested for mid-year 

should be used for the first year appraisal. 

In addition to the schematic presentation outlined, which follows, 

an appraisal in terms of the same categories that were developed in the 

initial proposal is recommended: These categories are: 

2. 1 Background (needs) 

2. 2 Objectives 

2. 3 Programs 

2.4 Demonstration 

2. 5 Dissemination 

2. 6 Implementation 

2. 7 Terminal considerations 

2.8 Management 

2. 9 Relationships 




FIRST YEAR: END OF YEAR REPORT 
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2. 10 Budget 

2.11 Facilities, equipment, and materials 
2. 12 Evaluation 

Appraisal of the ongoing program in terms of initial objectives 
and program operation is possible using this procedure. Some caution 

should be exercised, however, in terms of expecting a one-for-one re- 

» 

lationship. Indeed, it is reasonable, and in many cases desirable, to 
expect considerable deviation from project objectives to project opera- 
tion. These variations should not be ignored or glossed over because 
analyses of these differences can provide useful insights into project 
directions, allocations of time and personnel, and accomplishments. 

Rating scale: Also, a rating scale is suggested for many of the 

items. This scale would be a ten- point continuum: 



10 ' 


? 8 


7 ( 


0 


5 4 


3 : 


2 


1 


ol 



























The highest Average , The lo\ est 

rating , an un- sometimes , rating, an 

qualified yes yes and no unqualified 



no 

J 

The evaluator should use whatever number — from 10 to 0 — that best 
describes how well he believes the project is accomplishing the point 
being judged. 

2 . 1 Background 

2. 1. 1 To what extent is background material developed for 

the initial proposal adequate or inadequate? (The 
blank space is an invitation to rate the item using the 
ten- point scale mentioned above.) 




2 . 1 . 2 
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To what extent is new background needed? 
2. 1. 2. 1 What new evidences are needed? 



Objectives 

2. 2. 1 What objectives were developed in the initial proposal? 



2. 2. 2 To what extent are the objectives developed in the 
initial proposal still appropriate? /check one/ 

very much so 

^much 

somewhat 

little 

very little relationship 

2. 2. 3 Is reasonable progress being made toward stated ob- 
j ectives? 

2. 2. 3. 1 What evaluative devices are used to deter- 
mine this progress? 
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2. 2. 3. 1. 1 Are these satisfactory? 



Programs 

2. 3. 1 Are programs developing as envisaged in the initial 

proposal? (This question should not be interpreted as 
favoring a close relationship between programs given 
in the proposal and those that have developed during 
the operation. ) /_check one_/ 

very much so 

^much 

somewhat 

little 

very little relationship 

2.3.2 Are new programs needed now, as a result of opera- 
tional experience? ^yes no perhaps 

2. 3. 2. 1 If "yes, " what are the needs? 



2. 3. 3 



How would you rate at this time various program 
thrusts? (The rating might include (a) in terms of 
general and/or specific effectiveness, (b) compari- 
sons with other PACE projects, and (c) the general 
opinion of the program. ) 

2. 3. 3. 1 Overall program 

2. 3. 3. 2 Specific program (name) 

2. 3. 3. 3 Specific program (name) 

2. 3. 3. 4 Etc. 



Demonstration (only if applicable to your project) 



2. 4. 1 



Are the purposes of the demonstration compatible with 
objectives of the project? 
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2. 4. 2 Is it evident that serious thought has been given to 
demonstration? 

2.4.3. Are procedures established for screening prospective 
visitors into at least three categories: the casual ob- 

server, the interested educator (teacner), and the 
specialist in the program or process being demon- 
strated? 

2.4.4 Are there adequate safeguards against overexposure 
of children to visitors? 

2. 4. 5 Are teacher and school officials protected against ex- 
cessive use of time for visitors? 

2. 4. 6 Have ways been doveloped to use reactions of visitors 
to improve the program? 

Dissemination 

2. 5. 1 To what extent has an effective system been established 
for cataloging, interpreting and distributing information 
developed in the project? 

2. 5. 1. 1 What changes are suggested? 



2. 5. 2 Appraisal grid : By the end of the first six months and 

certainly by the end of the first year, a systematic ap- 
praisal of dissemination is in order — one that methodi- 
cally appraised audiences, media, and impact. Place 
in each square the number — 10 to 0 that best repre- 
sents the cegree of success in dissemination. Where 
the category is not applicable, place an "X. " 
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Audiences 



Board of Education 
Colleges and Universities 
Community 

Local Administrative Staff 
PACE Controlling Board 
Project Staff 
Specializing Groups 
State Department of 
Education 
Students 
Teachers 



Mediums 























































































































































































Informal Meet- 
inp-s 


In-service P: o- 
grams 


Newspapers 


Radio 


Regular Meet- 
ings 


Regular News- 
letter 


Special Bulletins 
(Memos) 


Special Stories 


Television 



2. 5. 3 Where are the strong points in dissemination? 



Why? 
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2. 5. 4 Where are the weak points? 



Why? 



2. 5. 5 What changes, if any, would seem in order? 



2. 5. 6 



Using the ten- point scale, 
disseminated in terms of: 



2. 5. 6. 1 
2 . 5 . 6 . 2 
2. 5. 6. 3 
2. 5. 6. 4 
2. 5. 6. 5 
2 . 5 . 6 . 6 



Clarity 

V alidity 

Pervasiveness 

Impact 

Timeliness 

Practicality 



judge the content of what is 

See Pace Manu.a.1 
for Project Ap- 
plicants and 
Grantees, Chap- 
ter Six, for ex- 
planation of these 
terms. 



2. 5. 7 What adjustments are needed for more effective dis- 

• r ^ 

semination, in terms of: 

2. 5. 7. 1 Time 
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2. 5. 7. 2 Money 



2. 5. 7. 3 Personnel 



2.5 7.4 Objectives 



2. 6 Implementation 

This criteria applies only if implementation is a thrust of the 
project. PACE programs featuring development, for example, should 
be expected to have quite a different approach to implementation from 
those focusing on moving a project from development action. 

2. 6. 1 What are the objectives of implementation? 



2. 6. 2 How do you rate general progress towards these ob- 
jectives? 



2. 6. 3 What adjustments need to be made to improve imple- 
mentation? (reminder checklist; not for rating. ) 

2. 6. 3. 1 No adjustment necessary; on target 

2. 6. 3. 2 Move more slowly 

2. 6. 3. 3 Move more rapidly 

2. 6. 3. 4 Need additional personnel 

2. 6. 3. 5 Need additional or different personnel skills 

2. 6. 3. 6 Need additional conceptual skills (ideas) 
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2. 6. 3. 7 Need to consider target audiences more 

carefully 

2. 6. 3. 8 Other 

2. 7 Terminal considerations 

Considerations should be given to terminal consider.ations at 
the end of the first year. And the word "creative" should typify this 
effort. 



2. 7. 1 To what extent are there evidences that the project 

staff is giving creative considerations to what happens 
when grant support ends? 



2. 7. 2 To what extent is planning evident that may facilitate 
continuation through other funding? (reminder check- 
list; not for rating. ) 

2.7.2. 1 Philanthropic foundations 

2. 7. 2. 2 Local school district 

2. 7. 2. 3 State Department 

2. 7. 2. 4 Other federal agency 

2. 7. 2. 5 Private industry 

2. 7. 2. 6 Other 



2. 8 Management 

2.8. 1 Have initial and/or r.-vi.sed rlescr iptions proven 

adequate? 

2.8. 1. 1 If not, would revision be useful in terms of 
time required? 
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2. 8. 2 To what degree are staff members familiar with their 
individual responsibilities in the overall program. 



2. 8. 3 To what extent would terms such as creative, innova- 
tive, open-minded, optimistic, intelligent, persever- 
ing, and humanly sensitive tend to characterize the 
project per sonnel? 



2.8.4 How effective are procedures for project personnel 
evaluation? 

2. 8. 4. 1 What improvements might be in order? — 



2. 8. 5 What new personnel needs have arisen? 



2.8.5. 1 What steps can be taken to meet these needs? 



2.8.6 Are procedures for intrastaff communications ade- 

. . . U IJ 

2. 8. 6. 1 If rating is low, what modifications should 
be made? 
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2. 8. 7 Are the planning techniques or procedures, such as 
PERT, checkpoints, etc. , being used to assist in the 
most effective use of time, resources, and personnel? 
2. 8. 7. 1 If rating is low, what can be done? 



2. 8. 8 Are adequate personnel and resources allocated for 
management functions? 

2. 8. 8. 1 If rating is low, what changes might be 
helpful? 



2. 8. 9 To what extent have staff in-service opportunities been 
available? (This would include conferences, meetings, 
workshops, etc.) 



2. 8. 10 Assuming an organizational chart exists, to what extent 
is it adequate? 



2. 9 Relationships 



2. 9. 1 In general, how would you judge relationships between 
the project staff and the following groups? (Use the 
ten-point rating scale.) 

2. 9. 1- 1 School officials 

2.9. 1.2 T(>achers 

2.9. 1 . 3 Students 
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2. 9. 1.4 
2. 9. 1. 5 
2. 9. 1. 6 
2. 9. 1. 7 
2 . 9 . 1. 8 
2. 9. 1. 9 
2.9.1. 10 



School Board 
Community at large 
State Department of Education 
Adjacent school district 
Mass media 

Universities and colleges 
Other 



2. 9. 2 What evaluative procedures are used to make deter- 
minations made in 2.9. 17__ 



2. 9. 3 Which groups listed in 2. 9. 1 have received most pro- 
ject time and resources?^ — 



2. 9. 3. 1 Should this allocation be changed? yes 

no perhaps 

2. 9. 3. 1. 1 If so, what strategy should be 
used in making the change? 



2. 9.4 If change is desirable, which groups should receive 
more attention? 

2. 9. 4.1 
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2. 9. 5 How are intrastaff relationships rated? 



2. 9. 6 How effective are inter staff relationships — those be- 
tween the PACE staff and the school system's staff? 



2.10 Budget 

2. 10. 1 Are clear records kept both by the district and the pro- 

j e ct? 

2. 10. 1.1 Is sufficient personnel allocated to this 

task? 



2. 10.2 Are bookkeeping procedures adequate? 



2 . 10. 3 



2 . 10. 4 



2 . 



10. 5 



Are programs realistically adjusted to budget? 

Has an audit been made or is it scheduled? 

Are budget records kept up-to-date? 

Is record-keeping achieved without undue friction? 



O 

ERIC 



2 . 10. 6 
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2. 10. 7 Are revisions in order following the first year? 
yes no perhaps 

2. 10. 7. 1 If revisions are needed, what? 



2.11 Facilities , equipment , and materials 

2. 11. 1 How adequate are basic facilities in terms of the pro- 
ject’s needs? 

2. 11. 2 Are facilities convenient in terms of primary liaisons 
for the project? 

2. 11.3 How appropriate are the basic project facilities in 

terms of style, decor, and furnishings? 

2. 11.4 How would you rate the quality of the materials selected 
in terms of how they were used? 

2. 11. 5 Were deliveries made on time? 

2.11.6 Were maintenance funds adequate? 

2. 11.7 Does the project have an efficient method for scheduling 
project personnel and students for use of available 
facilities and equipment? 

2. 11.8 What improvements should be made in facilities, equip- 
ment, and materials for the second year? 

2. 12 Evaluation 



2. 12. 1 Are there evidences that the PACE staff has paid seri- 
ous attention to evaluation? 
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2 . 12 . 2 



2 . 12. 3 



2. 12.4 



2 . 12. 5 



2 . 12 . 



2 . 12 . 



Is evaluation an integral part of the project? 



To what extent are evaluative procedures chosen in 
terms of the types of programs to be evaluated? (A 
low score would indicate that those evaluative proce- 
dures being used are inappropriate for what is being 
evaluated. ) 



How appropriate are the data 
2. 12.4. 1 Internal validity 
2. 12. 4. 2 External validity 

2.12.4.3 Reliability 

2.12.4.4 Objectivity 



in terms of: 

For explanations 
of these terms, 
see evaluation 
section (1.4, 2. 12.) 



Is a budget item visible for evaluation, or is it neces- 
sary to scrape together several vidds- and -ends to 
"prove" an evaluation sincerity? 

visible 

"odds- and- ends" accumulation 

Are evaluation procedures employed related to those 

outlined in the initial proposal? yes no 

somewhat 

2.12.6.1 If different, why? 



Is evaluation effectively related to program improve- 
ment? 
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2 . 12. 8 



What is the overall evaluation of the effectiveness of 
the evaluative procedures? 



H 



3. SECOND YEAR APPRAISAL 



The second year should find the project in full stride. In terms 
of operation, most projects should be shifting attention to emphasis 
from background, objectives, management, and budget to^ programs , 
relationships, demonstratic ', dissemination, implementation, and 
terminal considerations. We are talking about degrees of emphasis 
or a shift rather than a drastic change since most all phases are im- 
portant throughout the life of the project. Evaluation as feedback and 
guide is important throughout the project, but a.t the end of two years 
evaluation as judgment should take on increased importance. 

The instrument suggested for the second mid-year checkpoint 
and for the end of the second year is essentially the same one pro- 
posed for the first year. In this way, the self evaluation as well as the 
outside evaluation can have continuity. 

Six - Month's Report 

To deal with the possibility of some, or considerable, change 
in the project's nature and direction, a mid-year checkpoint a self- 
assessment — is suggested for every project. (See page 51.) The fol- 
lowing procedure is suggested as a method of bringing project develop- 
ment into manageable perspective at the mid-year checkpoint. 
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RO = revised objectives 



- 52 - 



On one large sheet, a "gestalt" of the entire program should be 
developed. This overview can be quite helpful in providing perspective 
for the staff as well as others. 

Second Y ear Appraisal 

A careful appraisal should be made at the end of the first year, 
from two perspectives: (1) a self-appraisal by the project's staff, and 

(2) an appraisal by an outside group. The results of these two apprai- 
sals then should be considered by the state ESEA Title III coordinator 
in developing his recommendations for the state advisory council. 

The same instrument should be used for both internal and exter- 
nal appraisals. The external appraisal group should make its evalua- 
tions independent of what the staff has done. 

The following two procedures are suggested for bringing the 
project into manageable perspective at the end of the second year. It 
is similar to the procedure recommended for the mid-year evaluation. 
The same schematic "gestalt" one-page picture suggested for mid-year 
should be used for the second year appraisal. 

In addition to the schematic presentation just outlined, an ap- 
praisal in terms of the same categories that were developed in the 
initial proposal is recommended: These categories are: 

3. 1 Background (needs) 

3. ? Objectives 
3. 3 Programs 
3. 4 Demonstration 
3. 5 Dissemination 



SECOND YEAR: END OF YEAR REPORT 
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3. 6 Implementation 

3. 7 Terminal considerations 

3. 8 Management 

3. 9 Relationships 

3. 10 Budget 

3. 11 Facilities,, equipment, and materials 
3. 12 Evaluation 

Appraisal of the ongoing program in terms of initial objectives 
and program operation is possible using this procedure. Some caution 
should be exercised, however, in terms of expecting a one-for-one re- 
lationship. Indeed, it is reasonable, and in many cases desirable, to 
expect considerable deviation from project objectives to project opera- 
tion. These variations should not be ignored or glossed over because 
analyses of these differences can provide useful insights into project 
directions, allocations of time and personnel, and accomplishments. 

Rating scale : Also, a rating scale is suggested for many of the 

items. This scale would be a ten- point continuum: 



10 


9 ! 


3 7 1 


6 


5 ^ 


3 : 


2 


1 


0 












\ ' 















1 - - — ■ * ' t ■ ■ ■ - „ 

The highest Average , The lowest 

rating , an un- sometime s rating , an 

qualified yes yes and no unqualified 



no 

The evaluator should use whatever number — from 10 to 0 — that best 
describes how well he believes the project is accomplishing the point 




being judged. 
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3. 1 Background 

3. 1. 1 To what extent is background material developed for 

the initial proposal and modified at the end of the first 
year adequate or inadequate? (The blank space is an 
invitation to rate the item using the ten- point scale 
mentioned on the preceding page. ) 

3.1.2 To what extent is new background needed? 

3. 1, 2. 1 What new evidences are needed? 



3. 2 Objectives 



3. 2. 1 



..^What objectives were developed in the initial proposal? 



3. 2. 2 To what extent are those objectives still appropriate? 
/check one/ 

very much so 

much 

_somewhat 

little 

very little relationship 
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3. 2. 3 Is reasonable progress being made toward stated ob- 
jectives? 

3. 2. 3. 1 What evaluative devices are used to deter- 
mine this progress?__ 



3. 2. 3. 1. 1 Are these satisfactory? 



3. 3 Programs 

3. 3. 1 Are programs developing as envisaged at the end of the 
first year? /pheck one7 

very much so 

much 

somewhat 

little 

very little relationship 

3. 3. 2 Are new programs needed now, as a result of two years 

of operational experience? ^yes no perhaps 

3. 3. 2. 1 If "yes, " what are the needs? 
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3.3.3 How would you rate at the end of two years the various 
program thrusts? (The rating might include (a) in 
terms of general and/or specific effectiveness, (b) 
comparisons with other PACE projects, and (c) the 
general opinion of the program. ) 

3. 3. 3. 1 Overall program (_ rating for the first 

year) 

3. 3. 3. 2 Specific program (name)_ 

3. 3. 3, 3 Specific program {name)__ 

3. 3. 3. 4 Etc. 

3.4 Demonstration (only if applicable to your project) 

3.4. 1 Are the purposes of the demonstration compatible with 
objectives of the project? 

3. 4. 2 Is it evident that serious thought has been given to 

demonstration? 

3. 4. 3 Are procedures established for screening prospective 
visitors into at least three categories; the casual ob- 
server, the interested educator (teacher), and the 
specialist in the program or process being demon- 
strated? 

3.4.4 Are there adequate safeguards against overexposure of 

children to visitors? 

3.4. 5 Are teacher and school officials protected against ex- 
cessive use of time for visitors? 

3. 4. 6 Have ways been developed to use reactions of visitors 
to improve the program? 

3. 5 Dissemination 

3. 5. 1 To what extent has an effective system been established 

for cataloging, interpreting and distributing informa- 
tion developed in the project? 

3. 5. 1. 1 What changes are suggested? 
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3. 5. 2 Appraisal grid : By the end of the second year, a sys- 

tematic appraisal of dissemination is in order — one 
that methodically appraises audiences, media, and im- 
pact. Place in each square the number — 10 to 0 — that 
best represents the degree of success in dissemination. 
Where the category is not applicable, place an "X. " 



Audiences 



Board of Education 
Colleges and Universities 
Community 

Local Administrative Staff 
PACE Controlling Board 
Project Staff 
Specializing Groups 
State Department of 
Education 
Students 
Teacher s 



Mediums 























































































































































































Informal Meet- 
ings 


■ ■ ■ ■ o ■ ■ - ■ 

In-service Pro- 
s^rams 


Newspapers 


Radio 


Regular Meet- 
ings 


Regular News- 
letter 


Special Bulletins 
fMemos^ 


Special Stories 


Television 



3. 5. 3 Where are the strong points in dissemination? 
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Why? 



3, 5. 4 Where are the weak points? 



Why? 



3. 5. 5 What changes, if any, would seem in order? (compare 
with first year appraisal.) 



3. 5. 6 



Using the ten- point scale, 
disseminated in terms of: 
3. 5. 6.1 Clarity 
3. 5. 6. 2 Validity 

3. 5. 6. 3 Pervasiveness 

3 . 5. 6 . 4 Impact 

3. 5. 6. 5 Timeliness 

3. 5. 6. 6 Practicality 



judge the content of what is 

See Pace Manual 
for Project Ap- 
plicants and 
Grantees, Chap- 
ter Six, for ex- 
planation of these 
I terms. 
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3. 5. 7 What adjustments at the end of two years are needed 
for more effective dissemination, in terms of: 

3. 5. 7.1 Time 



3. 5. 7. 2 Money 



3. 5. 7. 3 Personnel 



3. 5. 7. 4 Objectives 



3. 6 Implementation 

This criteria applies only if implementation is a thrust of the 
project. PACE projects featuring development, for example, should 
be expected to have quite a different approach to implementation from 
those focusing on moving a project from development to action. 

3.6.1 What are the objectives of implementation? 



3.6.2 What changes are evident at the end of two years as 
compared with the first year? 
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3. 6. 3 How do you rate general progress towards revised im- 
plementation objectives? 



3.6.4 What adjustments need to be made to improve imple- 
mentation? (reminder checklist; not for rating.) 

3.6.4. 1 No adjustments necessary; on target 

3. 6. 4. 2 Move more slowly 

3. 6. 4. 3 Mo^'^e more rapidly 

3. 6. 4. 4 Need additional personnel 

3. 6. 4. 5 Need additional or different personnel skills 

3. 6. 4. 6 Need additional conceptual skills (ideas) 

3. 6. 4. 7 Need to consider target audiences more 

carefully 
3. 6. 4. 8 Other 



3. 7 Terminal considerations 

Serious consideration should be given to terminal considera- 
tions at the end of the second year. And the word "creative" should 



typify this effort. 

3. 7. 1 To what extent are there evidences that the project 

staff is giving serious and creative consideration to 
what happens when grant support ends? 
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3.7.2 To what extent is planning evident that may facilitate 
continuation through other funding? (reminder check- 
list; not for rating. ) 

3.7.2. 1 Philanthropic foundations 

3. 7. 2. 2 Local school district 

3. 7. 2. 3 State Department 

3. 7. 2. 4 Other federal agency 

3. 7. 2. 5 Private industry 

3. 7. 2. 6 Other 



3. 8 



Management 



3. 8. 1 Have initial and/or revised job descriptions proven 
adequate? 

3. 8. 1. 1 If not, would revision be useful in terms of 
time required? 



3.8.2 To what degree are staff members familiar with their 
individual responsibilities in the overall program? 



3. 8. 3 To what extent would terms such as creative, innova- 
tive, open-minded, optimistic, intelligent, persevering, 
and humanly sensitive tend to characterize the project 
personnel? 



3.8.4 How effective are procedures for project personnel 
evaluation? 

3.8.4. 1 What improvements might be in order? 
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3.8.5 What new personnel needs have arisen during the 
second year? 



3.8.5. 1 What steps can be taken to meet these needs? 



3.8.6 Are procedures for intrastaff communications ade- 
quate? 

3.8.6. 1 If rating is low, what modifications should 
be made? 



3.8. 7 Are the planning techniques or procedures, such as 
PERT, checkpoints, etc. , being used to assist in the 
most effective use of time, resources, and personnel? 
3. 8. 7. 1 If rating is low, what can be done? 



3.8.8 Are adequate personnel and resources allocated for 
management functions? 

3. 8. 8. 1 If rating is low, what changes might be 

helpful? 



3. 8. 9 To what extent have staff in-service opportunities been 
available? (This would include conferences, meetings, 
workshops, etc. ) 
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3.8. 10 Assuming an organizational chart exists, to what ex- 
tent is it adequate? 



3. 9 Relationships 



3. 9. 1 



In general, how would you judge relationships between 
the project staff and the following groups? (Use the 
ten- point rating scale.) 



3.9.1. 1 
3. 9. 1.2 
3. 9. 1. 3 
3.9. 1.4 
3. 9. 1. 5 
3. 9. 1.6 
3. 9. 1. 7 
3. 9. 1. 8 
3. 9. 1.9 
3. 9. 1. 10 



School officials 

Teachers 

Students 

School Board 

Community at large 

State Department of Education 

Adjacent school district 

Mass media 

Universities and colleges 
Other 



3. 9. 2 What evaluative procedures are used to make determina- 
tions made in 3.9. 1? 



3. 9. 3 Which groups listed in 3. 9. 1 have received most pro- 
ject time and resources? 



ERLC 



3. 9. 3. 1 Should this allocation be changed? 

no uncertain 



_yes 



3. 9. 3. 1. 1 



If so, what strategy should be 
used in making the change? 
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3.9.4 If change is desirable, which groups should receive 
more attention? 

3.9.4. 1 Why? 



3. 9. 5 How are intrastaff relationships rated? 



3. 9. 6 How effective are inter staff relationships — those be- 
tween the PACE staff and the school system’s staff?_ 



3. 10 Budget 

3. 10. 1 Are clear budget records kept both-bythe district and 

the project? 



3. 10. L, 1 Is sufficient personnel allocated to this 
task? 



3. 10.2 Are bookkeeping procedures adequate? 



3. 10. 3 Are programs realistically adjusted to budget? 

3. 10.4 Has a second audit been made or is it scheduled? 
3. 10. 5 Are budget records kept up-to-date? 
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3. 11 



3. 12 



3. 10.6 Is record-keeping achieved without undue friction? 



3. 10. 7 Are revisions in order following the first year? 
^yes no perhaps 

3. 10. 7. 1 If revisions are needed, what? 



Facilities , equipment , and materials 



3. 11. 1 How adequate are basic facilities in terms of the pro- 
ject's needs? 

3. 11,2 Are facilities convenient in terms of primary liaisons 
for the project? 



3. 11.3 How appropriate are the basic project facilities in 
terms of style, decor, and furnishings? 

3. 11.4 How would you rate the quality of the materials selected 
for programs in terms of how they were used? 

3. 11,5 Were deliveries made on time? 



3. 11,6 Were maintenace funds adequate? 

3, 11, 7 Does the project have an efficient method for scheduling 
project personnel and students for use of available 
facilities and equipment? 

3. 11,8 What improvements should be made in facilities, equip- 
ment, and materials for the third year? 



Evaluation 



3. 12. 1 Are there evidences that the PACE staff is paying seri- 
ous attention to evaluation? 
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3. 12. 2 Are changes in evaluation evident between the first and 

the second year? ^yes no perhaps; please 

describe: 



3. 12. 3 Is evaluation an integral part of the project? 



3. 12.4 To what extent are evaluative procedures chosen in 
terms of the types of programs to be evaluated? (A 
low score would indicate that those evaluative proce- 
dures being used are inappropriate for what is being 
evaluated. ) 



3. 12. 5 



How appropriate are the data in 
3. 12.5. 1 Internal validity 
3. 12. 5. 2 External validity 

3.12.5.3 Reliability 

3.12.5.4 Objectivity 



terms of: 

For explanations 
of these terms, 
see evaluation 
section (1.4.2.12.) 



o 

ERIC 

hiMffliifftaiTi-Taaa 
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3. 12.6 



3. 12. 7 



3. 12. 



Is a budget item visible for evaluation, or is it neces- 
sary to scrape together several odas-and-ends to 
"prove" an evaluation sincerity? 

visible 

"odds-and-ends" accumulation 

Are evaluation procedures employed related to those 
outlined in the initial proposal and/or in the first year 

appraisal? yes no somewhat 

3.12.7.1 If different, why? 



Is evaluation effectively related to program improve- 
ment? 



What is the overall evaluation of the effectiveness of 
the evaluative procedures? 



3. 12. 9 



4. THIRD YEAR APPRAISAL 



The final year should show an increased accentuation toward 
demonstration, dissemination, implementation, terminal considera- 
tions, evaluation and budget. 

A human problem of future employment most likely will arise 
during the final year, and those involved with PACE projects should be 
frank and creative in dealing with this concern. Most PACE personnel 
are attracted to the project by its creative, innovative, and ambitious 
scope. This type of individual — likely to be more intelligent, critical, 
idealistic, restless, and daring than his more conventionally employed 
colleagues — is a vital element in public educ?>'"’'^n. We need many 
more individuals of this nature. Quoting from the first national study 
of ESEA Title III: 

"Within the field of professional education many dynamic, 
intelligent, creative, ambitious, and restless individuals can 
be found. They exist in every school system across the Nation 
and they can be a vital force in educational improvement. Too 
many of this group leave education because of low salaries and 
poor working conditions, to be sure, but probably more leave 
because of frustration and lack of challenge. 

PACE has become the natural home for this group. The 
special consultants and the director have been impressed with 
the enthusiasm and intelligence found among the project direc- 
tors. " 

In some cases, future employment will be no problem because 
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the PACE professionals were "loaned" frona the local school system, 
but this situation does not hold in many other instances. In any case, 
future professional moves should be approached openly, and early, in 
the third year. 

The final appraisal should vise several categories that were ap- 

I 

propriate for the first and second years, but some additional elements 
should be included, particularly points that have to do with wrapping 
up the project. 

Before moving into some suggested items, a cautionary note is 
raised about pyschologically wrapping up the project a few month® be- 
fore the actual terminal date. A strong and proper finish is completely 
in order. There may be a tendency for staff members, anticipating 
their future position, to sort of hope that things will finish themselves. 
Six - Month* s Report 

The following procedure is suggested as a method of bringing 
project development into manageable perspective at the mid-year 
checkpoint . 

On one large sheet, a "gestalt" of the entire program should be 
developed. (See page 71.) This overview can be quite helpful in pro- 
viding perspective for the staff as well as others. 

Third Year Appraisal 

A careful appraisal should be made at the end of the three year 
life of the project, from two perspectives: (1) a self-appraisal by the 



THIRD YEAR: SIX -MONTH REPORT 
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project's staff, and (2) an appraisal by an outside group. The results 
of these two appraisals then should be considered by the state ESEA 
Title III coordinator in developing his final report on the project. 

The same instrument should be used for both internal and ex- 
ternal appraisals. The external appraisal group should make its 
evaluations independent of what the staff has done. 

The following two procedures are suggested for bringing the 
project into final perspective. It is similar to the procedure recom- 
mended for the mid-year evaluation. 

In addition to the schematic presentation just outlined, an ap- 
praisal in terms of almost all of the categories that were developed in 
the init’al proposal is recommended. Categories suggested for the 
terminal report are: 

4. 1 Objectives 

4. 2 Programs 

4, 3 Demonstration 

4.4 Dissemination 

4. 5 Implementation 

4. 6 Terminal considerations 

4. 7 Management 

4. 8 Relationships 

4. 9 Budget 

4. 10 Facilities, equipment, and materials 
4. 11 Evaluation 

Rating scale : The same rating scale is suggested for the termi- 

nal report as for the other appraisals. This scale is a ten-point 
continuum: 




THIRD YEAR : END OF YEAR REPORT 
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10 



8 



The highest 
rating , an nn- 
qualifi'’‘^d yes 



Average , 
sometimes 
yes and no 




rating , an 

nnanalified 

no 



4. 1 Obiectives 

4, 1. 1> What objectives were developed in the initial proposal? 




4. 1. 2 To what extent are the objectives de'^loped in t e 
initial proposal still appropriate? /check one/ 

very much so 

much 

s omewhat 

__little 

^very little relationship 

4. 1. 3 Is reasonable progress being made toward stated ob 
jectives? 

4. 1. 3. 1 What evaluative devices are used to deter 
mine this progress? 



V 
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4. 1.3 1. 1 Are these satisfactory? 



4. 2 Programs 



4. 2. 1 Were programs developed as envisaged in the initial 
proposal? (This question should not be interpreted 
as favoring a close relationship between programs 
given in the proposal and those that have developed 
during the operation. ) Upheck one/ 

very much so 

much 

somewhat 

little 

^very little relationship 



4. 2. 2 Are new programs needed in the future as a result of 

the project experience? yes no perhaps 

4. 2. 2. 1 If "yes, " what are the needs? 



o 

ERIC 



4. 2. '3 What final rating would you give to the various pro- 
gram thrust? (The rating might include (a) in terms 
of general and/or specific effectiveness, (b) com- 
parisons with other PACE projects, and (c) the 
general opinion of the program. ) 

4. 2. 3. 1 Overall program (after three years of opera- 
tion) 

4. 2. 3. 2 Specific program (name) 

4. 2. 3. 3 Specific prograrrt. (name) 

4. 2. 3. 4 Etc. 



A 



Demonstration (only if applicable to your project) 



4, 3. 1 "Were the purposes of the demonstration compatible 
with objectives of the project? 

4. 3. 2 Is it evident that serious thought was given to demon- 
stration*? 

4. 3. 3 Were procedures established for screening prospec- 
tive visitors into at least three categories: the casual 

observer, the interested educator (teacher), and the 
specialist in the program or process being demon- 
strated? 

4.3.4 Were there adequate safeguards against overexposure 
of children to visitors? 

4. 3. 5 Were teacher and school officials protected against 
excessive use of time for visitors? 

4. 3. 6 Were ways developed to use reactions of visitors to 
improve the programs? 

Dis semination 



4.4. 1 To what extent was an effective system established for 
cataloging, interpreting, and distributing informa- 
tion developed in the project? 

4.4. 2 Appraisal grid : A systematic appraisal of dissemina- 

tion is in order — one that methodically appraises 
audiences, media, and impact. Place in each square 
the number — 10 to 0 — that best represents the degree 
of success in dissemination. Where the category is 
not applicable, place an "X." 



Audiences 



Board of Education 
Colleges and Universities 
Community 

Local Administrative Staff 
PACE Controlling Board 
Project Staff 
Specializing Groups 
State Department of 
Education 
Students 
Teachers 



Mediums 























































































































































































Informal Meet- 
ings 


In-service Pro- 
grams 


Newspapers 


Radio 


Regular Meet- 
ings 


Regular News- 
letter 


Special Bulletins 
(Memo s) 


Special Stories 


Television 



4 . 4 . 3 



Where were the 



strong points in dissemination? 



Why? 
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4. 4. 4 Where were the weak points? 



Why? 



4.4. 5 Using the ten- point scale, judge the contents of what is 
dissemination during the final year in terms of: 



4.4. 5. 1 
4.4. 5.2 
4. 4. 5.3 
4.4. 5.4 
4. 4. 5. 5 
4.4. 5. 6 



Clarity 

Validity 

Pervasiveness 

Impact 

T imeliness 

Practicality 



See Pace Manual 
for Project Ap- 
plicants and 
Grantees, Chap- 
ter Six, for ex- 
planation of these 



4. 5 Implementation 

This criteria applies only if implementation is a thrust of the 
project. PACE programs featuring development, for example, should 
be expected to have quite a different approach to implementation from 
those focusing on moving a project from development^ action. 

4. 5. 1 What were the objectives of implementation? 
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4. 5. 2 How would you rate general progress towards these ob 
jectives during the final year? 



4. 5. 3 What adjustments were needed to improve implementa- 
tion? (reminder checklist; not for rating. ) 

4. 5. 3. 1 No adjustments necessary; on target 
4.5. 3.2 Move more slowly 
4. 5. 3. 3 Move more rapidly 
4. 5. 3. 4, Need additional personnel 

4. 5. 3. 5 Need additional or different personnel skills 
4. 5. 3. 6 Need additional conceptual skills (ideas) 

4. 5. 3. 7 Need to consider target audiences more 
carefully 
4. 5. 3 . 8 Other 



4. 6 Terminal considerations 

Very serious consideration should be given to term.inal con 

siderations during the final year. And the words "creative' and ' per- 
sistent" should typlify this effort. 

4. 6. 1 To what extent were there evidences that the project 
staff gave serious and creative consideration to what 
happens when grant support ends?__ 



4. 6. 2 To what extent was planning evident that might facilitate 
continuation through other funding? (reminder check- 
list; not for rating. ) 

4. 6. 2.1 Philanthropic foundations 
4. 6. 2. 2 Local school district 
~ 4. 6. 2. 3 State Department 

4. 6. 2. 4 Other federal agency 
4. 6. 2. 5 Private industry 
4. 6. 2. 6 Other 
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Management 

4. 7. 1 To what degree were staff members familiar with 
their individual responsibilities in the overall pro- 
g r am? — 



4. 7. 2 How effective were procedures for personnel evalua- 
tion? 



4. 7. 3 What new personnel needs- arose during the last year? 



4. 7. 3. 1 What steps were taken to meet these needs? 



4. 7.4 Were procedures for intrastaff communications ade- 
quate? 

4. 7.4. 1 If rating is low, what modifications should 
be made? 



4. 7. 5 Were planning techniques or procedures, such as 

PERT, checkpoints, etc. , used to assist in the most 
effective use of time, resources, and personnel? 
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4. 7. 6 Were adequate personnel and resources allocated for 
management functions? 



4. 7. 7 To what extent were staff in-service opportunities 

available? (This would include conferences, meetings, 
workshops, etc. ) 



4.7.8 Assuming an organizational chart existed, to what ex- 
tent was it adequc’.te?_ — _ 



Relationships 



4.8. 1 



In general, how would you judge relationships between 
the project staff and the following groups? (Use the 
ten-point rating scale. ) 

4. 8. 1. 1 School officials 

4. 8 . 1 . 2 T eacher s 

4.8. 1.3 Students 

4. 8. 1. 4 School Board 

4. 8. 1.5 Community at large 

4. 8. 1.6 State Department of Education 

4.8. 1.7 Adjacent school district 

4.8. 1.8 Mass media 

4. 8. 1. 9 Universities and colleges 

4.8. 1 . 10 Other 



4. 8. 2 



What evaluative procedures were used to make deter- 
minations made in 4. 8. !?_ 



1 



ERLC 
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4. 8. 3 Which groups listed in 4. 8. 1 received most project 
time and resources? 



4.8.4 How would you rate intrastaff relationships during the 
third year?^ 



4. 8. 5 Were inter staff relationships effective — those between 
the PACE staff and the school system's staff?, 



4. 9 



B udget 
4. 9. 1 



Were clear budget records kept both by district and the 
project? — 



4. 9. 2 Were bookkeeping procedures adequate during the 
third year? 



4. 9. 3 Were programs realistically adjusted to budget? 

4. 9. 4 Was an audit made? 

4. 9. 5 Were budget records kept up-to-date? 

4. 9. 6 Was record-keeping achieved without undue friction? 
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4. 10 Facilities , equipment , and material^ 

4. 10. 1 How would you rate the quality of the materials 
selected in terms of how they were used? 

4. 10. 2 Were deliveries made on time? 

4. 10., 3 Does the project have an efficient method for scheduling 
project personnel and students for use of available 
facilities and equipment? 

4. 10.4 What ’mprovements should be made in facilities, equip- 
ment, and materials for the third year? 

4. 10. 5 What disposition is planned for the available facilities, 
equipment, and materials? 

4. 10. 5. 1 Are all details worked out, including legal 
aspects of transfer? 



4. 11 ■'^.valuation 

4. 11. 1 Did the PACE staff pay serious attention to evaluation 
during the final year? ^ 



4. 11. 2 Is evaluation an integral part of the project? 



4. 11. 3 To what extent are evaluative procedures chosen in 
terms of the types of programs to be evaluated? (A 
low score would indicate that those evaluative procc- 
d^ures being used are inappropriate for what is being 
evaluated. ) — — 
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4. 11. 4 



How appropriate are the dat 
4.1L4.1 Internalvalidity 

4.11.4.2 External validity 

4.11.4.3 Reliability 

4.1104.4 Objectivity 



in terms of: 

F or explanations 
of these terms, 
see evaluation 
section (1.4.2. 12.) 



4,11.5 Was a budget item visible for evaluation, or was it 
necessary to scrape together several odds-and-ends 
to "prove" an evaluation sincerity? 

vi s ible 

"odds-and-ends" accumulation 



4. 11.6 Was evaluation effectively related to program improve- 
ment during the third year? — 



4. 11. 7 What is the overall evaluation of the effectiveness of 
the evaluative procedures? 



5 . FINAL APPRAISAL 



This section is designed to assist further in the process of 
evaluating the final product, of considering what lessons have been 
learned from the three year experience, and of judging what learnings, 
if any, should be disseminated for the benefit of others 
5, 1 Final evaluation 

5, L 1 What would be the overall judgment of the project's 

effectiveness in terms of what it set out to do — the 
obj e ctive s ? 



5 2 Lessons learned 

5,2. 1 What positive lessons were learned? 
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5. 2. 2 What negative lessons were learned? 



5 . 3 Dissemination 

5. 3, 1 What project accomplishments v/ould seem '.ufficient.'-y 
significant to be wide’y disseminated? 

5. 3. 1,1 Research 
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5.3, 1,2 Instructional materials (such as new pro- 
grams for social studies, new materials for 
handicapped) 



5. 3, 1. 3 Skill development (such as reading, mathe- 
matics, vocational programs, etc . ) 
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5.3. 1.4 Process development (such as inquiry train- 
ing)^ 



5.3. 1.5 Orgranizational development (such as non- 
graded school, team teaching) 



5.3. 1.6 Pupil relations (such as guidance and coun- 
seling) 



5. 3. 1. 7 Community relations 



5. 3. 1.8 In-service programs 
ministrator s 



for teachers and/or ad- 
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5.3 1.9 Demonstration 



5.3. 1. 10 Dissemination 



5 3.1 11 Implementation (procedures for putting ideas 
into action) 



